CP-NAS: Child-Parent Neural Architecture Search for 1-bit CNNs
99
FIGURE 4.3
Performances of structures searched by ABanditNAS with different hyper-parameter values
λ.
savings, which is widely considered as one of the most efficient ways to perform computing
on embedded devices with low computational cost. In [199], the XNOR network is presented,
where the weights and inputs attached to the convolution are approximated with binarized
values. This efficiently implements convolutional operations by reconstructing the unbina-
rized filters with a single scaling factor. In [77], a projection convolutional neural network
(PCNN) is proposed to implement binarized neural networks (BNNs) based on a simple
back-propagation algorithm. [287] proposes Bayesian optimized 1-bit CNNs, taking advan-
tage of Bayesian learning to significantly improve the performance of extreme 1-bit CNNs.
Binarized models show advantages in reduction in computational cost and memory savings.
However, they suffer from poor performance in practical applications. There still remains a
gap between 1-bit weights/activations and full-precision counterparts, which motivates us
to explore the potential relationship between 1-bit and full-precision models to evaluate bi-
narized networks performance based on NAS. This section introduces a Child-Parent model
to efficiently search for a binarized network architecture in a unified framework.
The search strategy for the Child-Parent model consists of three steps shown in Fig. 4.4.
First, we sample the operations without replacement and construct two classes of subnet-
works that share the same architecture, i.e., binarized networks (child) and full-precision
networks (parent). Second, we train both subnetworks and obtain the performance indicator
of the corresponding operations by calculating the child network accuracy and the accuracy
TABLE 4.1
The performance of ABanditNAS with different search spaces on CIFAR10.
Architecture
# Search
Accuracy
# Params
Search Cost
Search
Space
(%)
(M)
(GPU days)
Method
ABanditNAS
7
97.13
3.0
0.09
Anti-Bandit
ABanditNAS
8
97.47
3.3
0.11
Anti-Bandit
ABanditNAS
9
97.52
4.1
0.13
Anti-Bandit
ABanditNAS
10
97.53
2.7
0.15
Anti-Bandit
ABanditNAS
11
97.66
3.7
0.16
Anti-Bandit